> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/ikawrakow/ik_llama.cpp/llms.txt
> Use this file to discover all available pages before exploring further.

# Quickstart

> Get ik_llama.cpp running in minutes on CPU or GPU

<Note>
  The fully supported backends are **CPU** (AVX2 or better, ARM NEON or better) and **CUDA**. ROCm, Vulkan, and Metal are available but not actively maintained in this fork.
</Note>

<Tabs>
  <Tab title="CPU">
    <Steps>
      <Step title="Clone the repository">
        ```bash theme={null}
        git clone https://github.com/ikawrakow/ik_llama.cpp
        cd ik_llama.cpp
        ```
      </Step>

      <Step title="Install dependencies (Debian/Ubuntu)">
        ```bash theme={null}
        apt-get update && apt-get install build-essential git libcurl4-openssl-dev curl libgomp1 cmake
        ```

        On other distros, install the equivalent packages for your package manager.
      </Step>

      <Step title="Build">
        ```bash theme={null}
        cmake -B build -DGGML_NATIVE=ON
        cmake --build build --config Release -j$(nproc)
        ```

        `-DGGML_NATIVE=ON` enables CPU-specific optimisations (AVX2, AVX-512, ARM NEON) for your machine. Omit it when cross-compiling.
      </Step>

      <Step title="Download a model">
        Download any GGUF model from HuggingFace. IQK quants from bartowski or ubergarm give the best quality/size tradeoff.

        ```bash theme={null}
        # Example: Qwen3 0.6B — ~400 MB, good for testing
        # https://huggingface.co/bartowski/Qwen_Qwen3-0.6B-GGUF
        ```
      </Step>

      <Step title="Start the server">
        ```bash theme={null}
        ./build/bin/llama-server --model /path/to/model.gguf --ctx-size 4096
        ```

        Open [http://127.0.0.1:8080](http://127.0.0.1:8080) in your browser to start chatting.
      </Step>
    </Steps>
  </Tab>

  <Tab title="GPU (CUDA)">
    <Steps>
      <Step title="Install prerequisites">
        Install the [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) from NVIDIA, then clone the repository:

        ```bash theme={null}
        git clone https://github.com/ikawrakow/ik_llama.cpp
        cd ik_llama.cpp
        ```
      </Step>

      <Step title="Build with CUDA support">
        ```bash theme={null}
        cmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=ON
        cmake --build build --config Release -j$(nproc)
        ```

        To target a specific GPU architecture (e.g. RTX 3090 = compute capability 8.6):

        ```bash theme={null}
        cmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
        ```
      </Step>

      <Step title="Download a model">
        Download a GGUF model from HuggingFace. For GPU inference, larger quants (IQ4\_KS, Q6\_K) are practical since VRAM is fast.
      </Step>

      <Step title="Start the server with GPU offload">
        ```bash theme={null}
        ./build/bin/llama-server --model /path/to/model.gguf --ctx-size 4096 -ngl 999
        ```

        `-ngl 999` offloads all layers to VRAM. Reduce the number if the model does not fit entirely in VRAM.

        Open [http://127.0.0.1:8080](http://127.0.0.1:8080) to start chatting.
      </Step>
    </Steps>

    <Note>
      FlashMLA (for DeepSeek models) requires an **Ampere or newer** NVIDIA GPU. For DeepSeek inference, also add `-mla 3 -fa` to your command.
    </Note>
  </Tab>
</Tabs>

<Tip>
  For the best quantization quality, look for models with IQK quants (`IQ4_KS`, `IQ5_K`, `IQ3_K`) or Trellis quants (`IQ2_KT`, `IQ3_KT`) on HuggingFace. These are exclusive to ik\_llama.cpp and outperform standard k-quants at the same bit-width.
</Tip>

## Next steps

<CardGroup cols={2}>
  <Card title="Building from source" icon="hammer" href="/building">
    Detailed build options for Windows, macOS, ROCm, and more.
  </Card>

  <Card title="GPU offloading" icon="microchip" href="/inference/gpu-offload">
    Fine-tune layer and tensor placement for maximum performance.
  </Card>

  <Card title="Quantization types" icon="layer-group" href="/quantization/overview">
    Understand IQK, Trellis, and how to pick the right quant.
  </Card>

  <Card title="Server reference" icon="server" href="/inference/server">
    All server options, API endpoints, and authentication.
  </Card>
</CardGroup>